Accelerating mini-batch SARAH by step size rules
نویسندگان
چکیده
StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective general nonconvex optimization, has received great attention because of its simple recursive framework updating stochastic gradient estimates. The performance SARAH significantly depends on the choice step size sequence. However, variants often manually select a best-tuned size, which is time consuming in practice. Motivated by this gap, we propose variant Barzilai-Borwein (BB) method, referred as Random (RBB) determine mini-batch setting, leading new method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly expectation strongly objective functions. Moreover, analyze complexity show it better than original method. To further confirm efficacy RBB MB-SARAH+-RBB incorporating into MB-SARAH + Numerical experiments standard data sets indicate our methods outperform or match state-of-the-art algorithms.
منابع مشابه
Distributed Mini-Batch SDCA
We present an improved analysis of mini-batched stochastic dual coordinate ascent for regularized empirical loss minimization (i.e. SVM and SVM-type objectives). Our analysis allows for flexible sampling schemes, including where data is distribute across machines, and combines a dependence on the smoothness of the loss and/or the data spread (measured through the spectral norm).
متن کاملMultiple Batch Sizing through Batch Size Smoothing
Batch sizing in different planning period is categorized as a classical problem in production planning, that so many exact & heuristic methods have been proposed to solve this problem, each of which considering various aspects of the original problem. The solution obtained from majority – e.g. MRP – is in this format that there may be some periods of idleness or each period should produce a...
متن کاملNested Mini-Batch K-Means
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested mini-batches, whereby data in a mini-batch at iteration t is automatically re...
متن کاملTurbocharging Mini-Batch K-Means
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested mini-batches, whereby data in a mini-batch at iteration t is automatically re...
متن کاملmultiple batch sizing through batch size smoothing
batch sizing in different planning period is categorized as a classical problem in production planning, that so many exact & heuristic methods have been proposed to solve this problem, each of which considering various aspects of the original problem. the solution obtained from majority – e.g. mrp – is in this format that there may be some periods of idleness or each period should produ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Information Sciences
سال: 2021
ISSN: ['0020-0255', '1872-6291']
DOI: https://doi.org/10.1016/j.ins.2020.12.075